{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7443df46",
   "metadata": {
    "origin_pos": 1
   },
   "source": [
    "# Parameter Management\n",
    "\n",
    "Once we have chosen an architecture\n",
    "and set our hyperparameters,\n",
    "we proceed to the training loop,\n",
    "where our goal is to find parameter values\n",
    "that minimize our loss function.\n",
    "After training, we will need these parameters\n",
    "in order to make future predictions.\n",
    "Additionally, we will sometimes wish\n",
    "to extract the parameters\n",
    "perhaps to reuse them in some other context,\n",
    "to save our model to disk so that\n",
    "it may be executed in other software,\n",
    "or for examination in the hope of\n",
    "gaining scientific understanding.\n",
    "\n",
    "Most of the time, we will be able\n",
    "to ignore the nitty-gritty details\n",
    "of how parameters are declared\n",
    "and manipulated, relying on deep learning frameworks\n",
    "to do the heavy lifting.\n",
    "However, when we move away from\n",
    "stacked architectures with standard layers,\n",
    "we will sometimes need to get into the weeds\n",
    "of declaring and manipulating parameters.\n",
    "In this section, we cover the following:\n",
    "\n",
    "* Accessing parameters for debugging, diagnostics, and visualizations.\n",
    "* Sharing parameters across different model components.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "41cbf7e2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:27:20.807089Z",
     "iopub.status.busy": "2023-08-18T19:27:20.806426Z",
     "iopub.status.idle": "2023-08-18T19:27:22.457089Z",
     "shell.execute_reply": "2023-08-18T19:27:22.456154Z"
    },
    "origin_pos": 3,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "293084ba",
   "metadata": {
    "origin_pos": 6
   },
   "source": [
    "(**We start by focusing on an MLP with one hidden layer.**)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9aa0461f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:27:22.461279Z",
     "iopub.status.busy": "2023-08-18T19:27:22.460607Z",
     "iopub.status.idle": "2023-08-18T19:27:22.494399Z",
     "shell.execute_reply": "2023-08-18T19:27:22.493545Z"
    },
    "origin_pos": 8,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 1])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = nn.Sequential(nn.LazyLinear(8),\n",
    "                    nn.ReLU(),\n",
    "                    nn.LazyLinear(1))\n",
    "\n",
    "X = torch.rand(size=(2, 4))\n",
    "net(X).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d0bb094",
   "metadata": {
    "origin_pos": 11
   },
   "source": [
    "## [**Parameter Access**]\n",
    ":label:`subsec_param-access`\n",
    "\n",
    "Let's start with how to access parameters\n",
    "from the models that you already know.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81e03323",
   "metadata": {
    "origin_pos": 12,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "When a model is defined via the `Sequential` class,\n",
    "we can first access any layer by indexing\n",
    "into the model as though it were a list.\n",
    "Each layer's parameters are conveniently\n",
    "located in its attribute.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f41ac20",
   "metadata": {
    "origin_pos": 14
   },
   "source": [
    "We can inspect the parameters of the second fully connected layer as follows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3c6fdb60",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:27:22.497996Z",
     "iopub.status.busy": "2023-08-18T19:27:22.497442Z",
     "iopub.status.idle": "2023-08-18T19:27:22.504291Z",
     "shell.execute_reply": "2023-08-18T19:27:22.503521Z"
    },
    "origin_pos": 16,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('weight',\n",
       "              tensor([[-0.1649,  0.0605,  0.1694, -0.2524,  0.3526, -0.3414, -0.2322,  0.0822]])),\n",
       "             ('bias', tensor([0.0709]))])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net[2].state_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc9e6e49",
   "metadata": {
    "origin_pos": 19
   },
   "source": [
    "We can see that this fully connected layer\n",
    "contains two parameters,\n",
    "corresponding to that layer's\n",
    "weights and biases, respectively.\n",
    "\n",
    "\n",
    "### [**Targeted Parameters**]\n",
    "\n",
    "Note that each parameter is represented\n",
    "as an instance of the parameter class.\n",
    "To do anything useful with the parameters,\n",
    "we first need to access the underlying numerical values.\n",
    "There are several ways to do this.\n",
    "Some are simpler while others are more general.\n",
    "The following code extracts the bias\n",
    "from the second neural network layer, which returns a parameter class instance, and\n",
    "further accesses that parameter's value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ba2da7b4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:27:22.507849Z",
     "iopub.status.busy": "2023-08-18T19:27:22.507181Z",
     "iopub.status.idle": "2023-08-18T19:27:22.513236Z",
     "shell.execute_reply": "2023-08-18T19:27:22.512406Z"
    },
    "origin_pos": 21,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.nn.parameter.Parameter, tensor([0.0709]))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(net[2].bias), net[2].bias.data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c10cf6c",
   "metadata": {
    "origin_pos": 24,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "Parameters are complex objects,\n",
    "containing values, gradients,\n",
    "and additional information.\n",
    "That is why we need to request the value explicitly.\n",
    "\n",
    "In addition to the value, each parameter also allows us to access the gradient. Because we have not invoked backpropagation for this network yet, it is in its initial state.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4c5f0ae9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:27:22.516723Z",
     "iopub.status.busy": "2023-08-18T19:27:22.516170Z",
     "iopub.status.idle": "2023-08-18T19:27:22.521606Z",
     "shell.execute_reply": "2023-08-18T19:27:22.520790Z"
    },
    "origin_pos": 27,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net[2].weight.grad == None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49d744bc",
   "metadata": {
    "origin_pos": 28
   },
   "source": [
    "### [**All Parameters at Once**]\n",
    "\n",
    "When we need to perform operations on all parameters,\n",
    "accessing them one-by-one can grow tedious.\n",
    "The situation can grow especially unwieldy\n",
    "when we work with more complex, e.g., nested, modules,\n",
    "since we would need to recurse\n",
    "through the entire tree to extract\n",
    "each sub-module's parameters. Below we demonstrate accessing the parameters of all layers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dab1b4b5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:27:22.525019Z",
     "iopub.status.busy": "2023-08-18T19:27:22.524380Z",
     "iopub.status.idle": "2023-08-18T19:27:22.530002Z",
     "shell.execute_reply": "2023-08-18T19:27:22.529195Z"
    },
    "origin_pos": 30,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('0.weight', torch.Size([8, 4])),\n",
       " ('0.bias', torch.Size([8])),\n",
       " ('2.weight', torch.Size([1, 8])),\n",
       " ('2.bias', torch.Size([1]))]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[(name, param.shape) for name, param in net.named_parameters()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fd29a2e",
   "metadata": {
    "origin_pos": 33
   },
   "source": [
    "## [**Tied Parameters**]\n",
    "\n",
    "Often, we want to share parameters across multiple layers.\n",
    "Let's see how to do this elegantly.\n",
    "In the following we allocate a fully connected layer\n",
    "and then use its parameters specifically\n",
    "to set those of another layer.\n",
    "Here we need to run the forward propagation\n",
    "`net(X)` before accessing the parameters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5b706636",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-08-18T19:27:22.533421Z",
     "iopub.status.busy": "2023-08-18T19:27:22.532786Z",
     "iopub.status.idle": "2023-08-18T19:27:22.541856Z",
     "shell.execute_reply": "2023-08-18T19:27:22.541011Z"
    },
    "origin_pos": 35,
    "tab": [
     "pytorch"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([True, True, True, True, True, True, True, True])\n",
      "tensor([True, True, True, True, True, True, True, True])\n"
     ]
    }
   ],
   "source": [
    "# We need to give the shared layer a name so that we can refer to its\n",
    "# parameters\n",
    "shared = nn.LazyLinear(8)\n",
    "net = nn.Sequential(nn.LazyLinear(8), nn.ReLU(),\n",
    "                    shared, nn.ReLU(),\n",
    "                    shared, nn.ReLU(),\n",
    "                    nn.LazyLinear(1))\n",
    "\n",
    "net(X)\n",
    "# Check whether the parameters are the same\n",
    "print(net[2].weight.data[0] == net[4].weight.data[0])\n",
    "net[2].weight.data[0, 0] = 100\n",
    "# Make sure that they are actually the same object rather than just having the\n",
    "# same value\n",
    "print(net[2].weight.data[0] == net[4].weight.data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ec93f84",
   "metadata": {
    "origin_pos": 38
   },
   "source": [
    "This example shows that the parameters\n",
    "of the second and third layer are tied.\n",
    "They are not just equal, they are\n",
    "represented by the same exact tensor.\n",
    "Thus, if we change one of the parameters,\n",
    "the other one changes, too.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a7b800f",
   "metadata": {
    "origin_pos": 39,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "You might wonder,\n",
    "when parameters are tied\n",
    "what happens to the gradients?\n",
    "Since the model parameters contain gradients,\n",
    "the gradients of the second hidden layer\n",
    "and the third hidden layer are added together\n",
    "during backpropagation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f0f0ad",
   "metadata": {
    "origin_pos": 40
   },
   "source": [
    "## Summary\n",
    "\n",
    "We have several ways of accessing and tying model parameters.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Use the `NestMLP` model defined in :numref:`sec_model_construction` and access the parameters of the various layers.\n",
    "1. Construct an MLP containing a shared parameter layer and train it. During the training process, observe the model parameters and gradients of each layer.\n",
    "1. Why is sharing parameters a good idea?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3987814f",
   "metadata": {
    "origin_pos": 42,
    "tab": [
     "pytorch"
    ]
   },
   "source": [
    "[Discussions](https://discuss.d2l.ai/t/57)\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "required_libs": []
 },
 "nbformat": 4,
 "nbformat_minor": 5
}